Compensation-free phase detection for phase detection autofocus devices

ABSTRACT

Enhanced phase detection for a PDAF sensor includes extracting pixel data from image data, the image data captured from an image capturing device having a phase detection autofocus (PDAF) sensor; extracting one or more features from the pixel data, including removing irrelevant pixel data; and determining a phase difference between the one or more features.

BACKGROUND

Modern image capturing devices, such as digital cameras, often includeauto focus (AF) to aid the image capturing device to focus upon anintended object. There are a variety of different techniques forperforming autofocus such as, for example, phase detection autofocus(PDAF). Additionally, image capturing devices can include image sensorsto assist with autofocusing an image. Some image capturing devices canuse particular types of sensors. For example, some image capturingdevices use PDAF sensors to perform autofocus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 sets forth a block diagram of an example system for providingenhanced phase detection in an image capturing device accordance withsome implementations of the present disclosure.

FIG. 2 sets forth a diagram of an example system for providing enhancedphase detection in an image capturing device with a lens at differentpositions to assist with phase detection autofocus in accordance withsome implementations of the present disclosure.

FIG. 3 sets forth a diagram of an example system for providing enhancedphase detection in an image feature domain using an image capturingdevice having phase detection autofocus devices in accordance with someimplementations of the present disclosure.

FIG. 4 sets forth a diagram of another example system for providingenhanced phase detection in an image capturing device having phasedetection autofocus devices in accordance with some implementations ofthe present disclosure.

FIG. 5 sets forth a flow chart illustrating another example method ofproviding enhanced phase detection in an image capturing device havingphase detection autofocus devices in accordance with someimplementations of the present disclosure.

FIG. 6 sets forth a flow chart illustrating an example implementation ofremoving the irrelevant pixel data in accordance with someimplementations of the present disclosure.

FIG. 7 sets forth a flow chart illustrating an example implementation ofextracting pixel data from image data in accordance with someimplementations of the present disclosure.

FIG. 8 sets forth a flow chart illustrating an example implementation ofmatching features of the pixel data in accordance with someimplementations of the present disclosure.

DETAILED DESCRIPTION

As previously mentioned, autofocusing can be performed using phasedetection autofocus (PDAF). PDAF is enabled by ‘focus pixels.’ Focuspixels are pixels on a sensor grid and are used for focusing rather thanon capturing the image. These focus pixels can come in pairs (e.g., aleft pixel and a right pixel), are positioned relatively close to eachother on a sensor such that one pixel can receive light from the left(or top) part of a lens while the other pixel in the pair receives lightfrom the opposing side (right or bottom) of the lens. (There are avariety of types of PDAF sensors such as, for example, partially maskedPDAF sensors. Some PDAF sensors may have a variety of types of designssuch as, for example, a PDAF sensor that has a micro-lens to separatelight.) Because these pixels are positioned near to each other, theamount of light they each receive should be approximately the same. Anydifference in the amount of light received between the pair of pixelscan be used to determine the image is “out of focus.” Said differently,if an image or signal produced from left pixel data and an image signalproduced from right pixel data is different, a phase difference can bedetermined between the different signals. The detected phase differencescan be used to perform autofocus. However, determining an accurate andcorrect phase difference can be compromised when irrelevant data isincluded in the image. Irrelevant data is data that does not form afeature intended to be captured in the image. An example of irrelevantdata may be foliage in the periphery of a scene where the primaryfeature to be captured is a large flower. While the foliage may alwaysbe captured in an image of the larger flower, the focal point isintended to be the flower. To that end, autofocus should ignore thefoliage and instead utilize on the large flower as the point or featureon which to drive the autofocus mechanism. Irrelevant data may also benoise in the pixel data that exists as a result of errors inpreprocessing, errors in image sensors, variations in lens geometry, andother types of noise as will occur to readers of skill in the art.

Accordingly, implementations in accordance with the present disclosureprovide a mechanism for enhanced phase detection for a PDAF sensor. Inan implementation, pixel data is extracted from image data that iscaptured from an image capturing device having a PDAF sensor. One ormore features are extracted from the pixel data that is extracted fromthe image data where irrelevant pixel data is removed from the pixeldata extracted from the image data. A phase difference is determinedbetween the one or more features of the pixel data extracted from theimage data.

An implementation is directed to a method of providing enhanced phasedetection for a PDAF sensor is disclosed. The method includes extractingpixel data from image data, the image data captured from an imagecapturing device having a PDAF sensor. The method also includesextracting one or more features from the pixel data extracted from theimage data, where irrelevant pixel data is removed from the pixel dataextracted from the image data. The method also includes determining aphase difference between the one or more features of the pixel dataextracted from the image data.

In some implementations, the method also includes filtering the pixeldata extracted from the image data to remove the irrelevant pixel data.In some implementations, extracting the pixel data from the image dataincludes extracting left pixel data and right pixel data from the imagedata. In some implementations, extracting the one or more features fromthe pixel data extracted from the image data includes extracting afeature from the left pixel data and a feature from the right pixel dataprior to determining the phase difference. In some implementations,determining the phase difference includes determining the phasedifference between a feature extracted from left pixel data and afeature extracted from right pixel data in an image feature domain.

In some implementations, the method also includes matching the one ormore features of the pixel data extracted based on the phase difference.In some implementations, the method also includes performing left-rightmatching between a feature extracted from left pixel data and a featureextracted from right pixel data in an image feature domain.

Another implementation is directed to an apparatus for providingenhanced phase detection. The apparatus comprises a computer processor,a computer memory operatively coupled to the computer processor, thecomputer memory having disposed therein computer program instructionsthat, when executed by the computer processor, cause the apparatus toextract pixel data from image data, the image data captured from animage capturing device having a PDAF sensor. The computer programinstructions also cause the apparatus to extract one or more featuresfrom the pixel data extracted from the image data, where irrelevantpixel data is removed from the pixel data extracted from the image data.The computer program instructions also cause the apparatus to determinea phase difference between the one or more features of the pixel dataextracted from the image data.

In some implementations, the computer program instructions cause theapparatus to filter the pixel data extracted from the image data toremove the irrelevant pixel data. In some implementations, extractingthe pixel data from the image data also includes extracting left pixeldata and right pixel data from the image data. In some implementations,extracting the one or more features from the pixel data extracted fromthe image data also includes extracting a feature from the left pixeldata and a feature from the right pixel data prior to determining thephase difference. In some implementations, determining the phasedifference also includes determining the phase difference between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain.

In some implementations, the computer program instructions cause theapparatus to carry out matching the one or more features of the pixeldata extracted based on the phase difference. In some implementations,the computer program instructions cause the apparatus to carry outperforming left-right matching between a feature extracted from leftpixel data and a feature extracted from right pixel data in an imagefeature domain.

Yet another implementation is directed to a computer program product forproviding enhanced phase detection. The computer program product isdisposed upon a computer readable medium and comprises computer programinstructions that, when executed, cause a computer to extract pixel datafrom image data, the image data captured from an image capturing devicehaving a PDAF sensor. The computer program instructions also cause thecomputer to extract one or more features from the pixel data extractedfrom the image data, where irrelevant pixel data is removed from thepixel data extracted from the image data. The computer programinstructions also cause the computer to determine a phase differencebetween the one or more features of the pixel data extracted from theimage data.

In some implementations, the computer program instructions also causethe computer to filter the pixel data extracted from the image data toremove the irrelevant pixel data. In some implementations, extractingthe pixel data from the image data also includes extracting left pixeldata and right pixel data from the image data. In some implementations,extracting the one or more features from the pixel data extracted fromthe image data also includes extracting a feature from the left pixeldata and a feature from the right pixel data prior to determining thephase difference. In some implementations, determining the phasedifference also includes determining the phase difference between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain.

In some implementations, the computer program instructions also causethe computer to carry out matching the one or more features of the pixeldata extracted based on the phase difference. In some implementations,the computer program instructions also cause the computer to carry outcarrying out the steps of performing left-right matching between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain.

Implementations in accordance with the present disclosure will bedescribed in further detail beginning with FIG. 1 . Like referencenumerals refer to like elements throughout the specification anddrawings. FIG. 1 sets forth a block diagram of an example system 100 forproviding enhanced phase detection in accordance with someimplementations of the present disclosure. The example system 100 ofFIG. 1 can be implemented in a computing device such as a mobile deviceincluding a smart phone or tablet, a laptop computer, digital camera,and so on. More specifically, system 100 of FIG. 1 can be implemented inan image capturing device, (e.g., a camera), 110 that includes at leasta lens 123 for capturing an image. The image that is captured by thelens 123 can include both visible (RGB) light and infrared (IR) light.It should be noted that additional types of light can be captured by thelens 123, but for purposes of example, visible and IR are discussedherein.

Although the image capturing device 110 has been described as a digitalcamera or the like, it is noted that the image capturing device 110 caninclude, for example, a computer, a gaming device, a handheld device, aset-top box, a television, a mobile phone (e.g., smart phone), or atablet computer. For purposes of example herein, the example device 110is described as an image capturing device 110, which for example is acamera. Accordingly, the image capturing device 110 includes a processor102, a memory 104, a storage 106, one or more input devices 108, acamera module 111, and one or more output devices 119. The imagecapturing device 110 can also optionally include an input driver 112 andan output driver 114. It is understood that the image capturing device110 can include additional components not shown FIG. 1 .

The camera module 111 includes the lens 123 described above, a filter124, and a sensor 125 such as, for example, a phase detection autofocus(PDAF) sensor that can provide enhanced phase detection as describedherein. The PDAF sensor 125 can be used, in conjunction with the lens123, to focus on an object of interest. The PDAF sensor 125 can be animage sensor equipped with phase detection (PD). The PD pixels arespecially manufactured pixels distributed uniformly on an image sensor(e.g., the PDAF sensor 125) with a certain percentage of occupation. Inone aspect, left PD pixels and right PD pixels receive light from theleft and right sides of the lens 123, respectively. Phase differencesgenerated between left PD pixels and right PD pixels can then be usedfor autofocusing the image capturing device 110, specifically, the lens123, for instant (or near-instant) auto focus. It should be noted, asused herein, any reference to “left pixel” or “right pixel” may alsoinclude or reference “top pixel” or “bottom pixel.” That is, anyreference to “left” or “right” may in general be referencing a firstside or a second side. Thus, a first side can be a left side and asecond side can be a right side. Alternatively, a first side can be atop side and a second side can be a bottom side. In another variations,a first side can include both a left side and top side and the secondside can include both a right side and a bottom side. Thus, reference toonly “left pixel” or “right pixel” such as, for example “left pixeldata” and “right pixel data” is used by way of example only.Accordingly, in some implementations, “left pixel” (or left pixel data)and “right pixel” (or right pixel data) can include or be replaced with“top pixel” (or top pixel data) and “bottom pixel” (or bottom pixeldata) For example, in some implementations, PD pixels can receive lightfrom a first side (e.g., a left side, top side, or both) and an oppositeside (e.g., a right side, bottom side, or both) of the lens.

Thus, the image capturing device 110 uses the components of the cameramodule 111, such as the lens 123 and the PDAF sensor 125, to separateleft and right light rays though the lens 123 in order to capture or‘sense’ left and right images. The left and right images can be comparedto each other to determine a difference (e.g., a phase difference) inthe position of the left and right images on the PDAF sensor 125. Thedifference can be used to determine a shift (a phase shift) of a cameralens 123 for autofocus.

It should be noted that filter 124 may be included in the cameral module111, the processor 102, or a combination of both the camera module 111and the processor 102. FIG. 1 depicts the filter 124 in both the cameramodule 111 and the processor 102 by way of example only. In one aspect,the filter 124 can be hardware, software, or both. In someimplementations, the filter 124 can operate as an application/softwaresolution executed by the processor 102 to perform feature (or edge)extraction and to remove irrelevant pixels for increasing and yieldingmore efficient phase detection results. In some implementations, thefilter 124 can operate as an application/software solution executed bythe processor 102, which may be internal or external to the imagecapturing device. Again, other configurations can be designed forimplementing and using filter 124.

In various alternatives, the processor 102 can include a centralprocessing unit (CPU), a graphics processing unit (GPU), and even anaccelerated processing units (APU), a CPU, GPU, and APU located on thesame die, or one or more processor cores where each processor core canbe a CPU, a GPU, or an APU. The process 102 can include filter 124. Invarious alternatives, the memory 104 is be located on the same die asthe processor 102, or is located separately from the processor 102. Thememory 104 includes a volatile or non-volatile memory, for example,random access memory (RAM), dynamic RAM, or a cache. An image signalprocessor (ISP) can be included in the processor 102 to perform enhancedphase detection image signal processing in the PDAF sensor 125 asdescribed in more detail below. Alternatively, the ISP can be includedin the APD 116 or as a separate processing unit (not shown). That is,although the location of the ISP is not specifically shown, it canreside separately from, or be integrated within the processor 102 or APD116. The storage 106 includes a fixed or removable storage, for example,a hard disk drive, a solid state drive, an optical disk, or a flashdrive.

The input devices 108 can include, without limitation, a keyboard, akeypad, a touch screen, a touch pad, a detector, a microphone, anaccelerometer, a gyroscope, a biometric scanner, or a network adapter(e.g., a wireless local area network card for transmission and/orreception of wireless IEEE 802 signals). The output devices 119 caninclude, without limitation, a display, a speaker, a printer, a hapticfeedback device, one or more lights, an antenna, or a network adapter(e.g., a wireless local area network card for transmission and/orreception of wireless IEEE 802 signals).

The input driver 112 communicates with the processor 102 the inputdevices 108, and the lens 123, and permits the processor 102 to receiveinput from the input devices 108 and the lens 123. The output driver 114communicates with the processor 102 and the output devices 110, andpermits the processor 102 to send output to the output devices 119.

It is noted that the input driver 112 and the output driver 114 areoptional components, and that the image capturing device 110 operates inthe same manner if the input driver 112 and the output driver 114 arenot present. The output driver 114 includes an accelerated processingdevice (APD) 116 which is coupled to a display device 118. In oneaspect, the display device 118 can be configured to provide a previewimage prior to capturing an image. The display 118 can comprise varioustypes of screens and can implement touch sensitive technologies.

The APD is configured to accept compute commands and graphics renderingcommands from processor 102, to process those compute and graphicsrendering commands, and to provide pixel output to display processor 102for display. In one aspect, the APD 116 includes one or more parallelprocessing units configured to perform computations in accordance with asingle-instruction-multiple-data (SIMD) paradigm that can assist with orbenefit from the enhanced phase detection using the PDAF sensor 125.Thus, although various functionality is described herein as beingperformed by or in conjunction with the APD 116, in variousalternatives, the functionality described as being performed by the APD116 is additionally or alternatively performed by other computingdevices having similar capabilities that are not driven by a hostprocessor (e.g., processor 102) and configured to provide graphicaloutput to a display device 118. For example, it is contemplated that anyprocessing system that performs processing tasks in accordance with aSIMD paradigm can be configured to perform the functionality describedherein. Alternatively, it is contemplated that computing systems that donot perform processing tasks in accordance with a SIMD paradigm performsthe functionality described herein. The image signal processingdescribed herein can also be performed by the APD 116.

In some implementations, the image capturing device 110 uses thecomponents of the camera module 111 such as, for example, the lens 123and the PDAF sensor 125 to extract left pixel data and right pixel datafrom the image data. In some implementations, the image capturing device110 uses the components of the camera module 111, such as the lens 123and the PDAF sensor 125, to extract a feature from the left pixel data(e.g., left phase detection pixels) and a feature from the right pixeldata (e.g., right phase detection pixels) prior to determining a phasedifference. In some implementations, the image capturing device 110 usesthe components of the camera module 111 such as, for example, the lens123 and the PDAF sensor 125 to determine the phase difference between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain. The phrase ‘image featuredomain’ generally refers to an area of an image that includes data thatforms an intended feature of the image. More specifically, the imagefeature domain comprises both the left pixel data and right pixel datathat includes an intended feature of the image. A single large flower,for example, can be included in an image feature domain of an image thatincludes nothing else but green foliage. The image feature domaingenerally includes a feature that is to be extracted from left pixeldata and a feature extracted from right pixel data, where irrelevantpixels are removed from the extracted features, and a true phasedifference can be determined without the irrelevant pixel datacompromising the phase difference determination.

For further explanation, FIG. 2 sets forth a diagram of an examplesystem for providing enhanced phase detection in an image capturingdevice with a lens at different positions to assist with phase detectionautofocus in accordance with some implementations of the presentdisclosure. That is, FIG. 2 depicts component an capturing device with alens at different positions such as, for example, position 1, position2, and position 3 to assist with phase detection autofocus in accordancewith some implementations of the present disclosure.

For example, the image capturing device 110 can be directed toward atarget object such as, for example, object 202. Position 1, position 2,and position 3 each represent a different lens position of the lens 123from the PDAF sensor 125 (e.g., PDAF image sensor) in the imagecapturing device 110 in relation to an object 202 for capturing an imageof the object 202. As depicted, position 1 and position 3 depict a phasedifference while position 2 depicts zero phase difference. Phasedifference, as used herein, can be the phase distance (PD) pixeldistance between a left image (formed by left PD pixels) and a rightimage (formed by right PD pixels). When there is a change in the lensposition of lens 123 for focusing, the phase difference changes. A largephase difference indicates a larger extent of lens defocus, such as, PD1of position 1 and PD3 of position 3. It should be noted the PD1 and PD3are arbitrary value/distance used herein for illustration purposes onlyto depict a phase difference of a certain value. PD 2 is used to depict“zero” phase difference or representing an “in-focus” image of theobject 202. When the phase difference turns to zero such as, forexample, as depicted in PD2 of position 2, the lens 123 can be at an“in-focus” position and can generate a focused image of the targetobject 202.

Using the lens 123 and the PDAF sensor 125 of the image capturing device110 of FIG. 1 , the autofocus functionality of the image capturingdevice 110 automatically moves or adjusts the lens 123 closer, asdepicted in position 1, or moving the lens 123 farther away from thePDAF sensor 125, as depicted in position 3. In this way, left light rays204 and right light rays 206 that reflect off the item or object 202that the image capturing device 110 is attempting to capture strikes ormakes contact at an identical point (e.g., a focal point) such as, forexample, locations 208 of position 1, 210 of position 2, and 212 ofposition 3 on the PDAF sensor 125 as they travel through the lens 123.

As illustrated, in position 3, the left light rays 204 and right lightrays 206 pass through the lens 123 and strike the PDAF sensor 125 atlocations 212. Here, the lens 123 is too close to the object in relationto the PDAF sensor 125 creating a phase difference between the left andright images, as show on graph 220 that depicts the left light rays 204and right light rays 206. The phase difference causes the object 202 tobe out of focus.

In position 2, the left light rays 204 and right light rays 206 passthrough the lens 123 and strike the PDAF sensor 125 at location 210,which is the intended focal point or “in-focus.” Here, the lens 123positioned at an appropriate distance to the object in relation to thePDAF sensor 125 and the lens 123 and correctly focused therebyeliminating any phase difference such as, for example, PD1 is equal tozero, as further depicted in graph 222 that depicts the left light rays204 and right light rays 206. It should be noted that graph 222 depictsthe left light rays 204 and right light rays 206 merged together showingan in-focus image.

In position 1, the left light rays 204 and right light rays 206 passthrough the lens 123 and strike the PDAF sensor 125 at location 208.Here, the lens 123 is too far from the object in relation to the PDAFsensor 125 creating another phase difference between the left and rightimages. The phase difference is shown on graph 224 which depicts theleft light rays 204 and right light rays 206. The phase differencecauses the object 202 to be out of focus, as further depicted in graph220. The phase difference at location 208 of position 3 and the phasedifference at location 212 of position 1 between locations where theleft light rays 204 and right light rays 206 strike the sensor 125 isthe phase shift. The phase shift can be used to determine the autofocusphase shift and direction for the image capturing device 110.

For further explanation, FIG. 3 sets forth a block flow diagram of anexample system for providing enhanced phase detection in an imagefeature domain using an image capturing device having phase detectionautofocus devices in accordance with some implementations of the presentdisclosure. As shown, the various blocks of functionality are depictedwith arrows designating the blocks’ relationships with each other and toshow process flow. Additionally, descriptive information is also seenrelating each of the functional blocks of FIG. 3 . As will be seen, manyof the functional blocks can also be considered “modules” offunctionality, in the same descriptive sense as has been previouslydescribed in FIG. 1 . With the foregoing in mind, the blocks can also beincorporated into various hardware and software components of a systemfor providing enhanced phase detection in an image feature domain usingan image capturing device having phase detection autofocus devices inaccordance with the present invention. Many of the functional blocks canexecute as background processes on various components, either indistributed computing components, or on the user device, or elsewhere.

Starting at image 310, the image 310 (e.g., a full raw image that hascontains minimally processed data) can be captured by the imagecapturing device 110 of FIG. 1 . The image capturing device 110 extractsleft PD pixels 320 and right PD pixels 324 from the image data 310(e.g., the full raw image). It should be noted that at this point, theleft pixel data (or top pixel data) and the right pixel data (or bottompixel data) can have unbalanced signal levels. For example, the leftpixel data are brighter at the right side of the image and right pixeldata is brighter at the left side. Because of the unbalanced nature ofthe signal levels, a matching operation or a left/right gaincompensation per pixel is required to bring the left/right signal levelsinto balance.

However, as illustrated herein, the image capturing device 110, performsa pre-processing operation 330 (e.g., a feature extractionpre-processing operation), by executing a feature extraction operationof the left PD pixels 320 and right PD pixels 324 prior to performing amatching operation in an image feature domain. One or more irrelevantportions (imperfect compensation, flat areas, horizontal lines, etc.),such as irrelevant pixel data 322 and irrelevant pixel data 326, can befiltered using filter 124 and removed from the left PD pixels 320 andright PD pixels 324 extracted from the image data 310 in the imagefeature domain. In one aspect, the irrelevant pixel data 322 andirrelevant pixel data 326 can be low-contrast or low-textured images orsmall, irrelevant objects. Also, the irrelevant pixels can be removed toeliminate any bias in performing a matching operation.

By extracting features and removing the irrelevant pixel data 322 andirrelevant pixel data 326, matching operations are improved, especiallyfor low-contrast, low-textured or small object scenes. Also, any need toperform a left/right gain compensation per pixel (e.g., phase differencecompensation) is eliminated. The extracted features in a left featureimage 340 and a right feature image 342 can be used to perform aleft/right matching operation to determine and learn a phase differencebetween the left feature image 340 and the right feature image 342. Atrue phase difference 350 can be determined from the left feature image340 and the right feature image 342.

For further explanation, FIG. 4 sets forth a block flow diagram of anexample system for providing enhanced phase detection in an imagefeature domain using an image capturing device having phase detectionautofocus devices in accordance with some implementations of the presentdisclosure. As will be seen, many of the functional blocks can also beconsidered “modules” of functionality, in the same descriptive sense ashas been previously described in FIGS. 1-4 . Repetitive description oflike elements employed in other implementations described herein isomitted for sake of brevity.

In one aspect, the image capturing device 110 of FIG. 1 can capture 410image data (e.g., a full raw image). For example, the PDAF sensor 125 ofFIG. 1 processes the image data captured through the lens 123. In oneaspect, processor 102 can be an image processor capable of working withthe PDAF sensor 125 to extract image data and perform enhanced phasedetection autofocus operations as described herein.

In an additional aspect, the PDAF sensor 125, in association withprocessor 102, extracts 420 the phase detection pixels (e.g., PD pixelextraction) from the image data where the image data is captured fromthe image capturing device 110. In one aspect, the PDAF sensor 125 andthe processor 102 can also execute instructions for calculating colorvalues for phase detection pixels and for image generation based onphase detection pixel values and imaging pixel values.

The image capturing device 110 of FIG. 1 , using the PDAF sensor 125, inassociation with processor 102, extracts 430 features from the phasedetection pixels (e.g., from left PD pixels and right PD pixels) as apre-processing operation. As part of the feature extraction, the filter124, in association with processor 102, filters the phase detectionpixels extracted from the image data to remove the irrelevant pixeldata.

By removing the irrelevant pixel data and filtering the phase detectionpixels, the PDAF sensor 125, in association with processor 102, providesleft pixel data and right pixel data (or top pixel data and bottom pixeldata) without the irrelevant pixel data that can bias a phase differencedetermination. The extracting can also include extracting a feature fromthe left pixel data and a feature from the right pixel data prior todetermining the phase difference. In one aspect, the feature extractioncan be performed using high/band-pass filters (e.g., a 1-dimensionalhigh/band-pass filter) to remove the irrelevant pixels such as, forexample, the flat areas and horizontal lines.

Subsequent to the pre-processing operation, the PDAF sensor 125, inassociation with processor 102, determines a phase difference between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain and match the one or morefeatures of the left pixel data and a feature extracted from right pixeldata based on the phase difference. More specifically, the PDAF sensor125, in association with processor 102, performs left-right (L/R)matching 440 between a feature extracted from left pixel data and afeature extracted from right pixel data in an image feature domain.

As depicted in graph 450, a true, unbiased phase difference isdetermined such as, for example, a phase difference of “3.” It should benoted that graph 450 depicts a degree of similarity on the y-axis and aphase difference on the x-axis. The values of graph 450 are arbitraryvalues and depicted for illustrated purposes only.

Thus, by applying the operations described herein such as, for example,in FIGS. 1-4 , the need of using a per-pixel gain compensation operationis eliminated thereby providing enhanced computing efficiency and costsavings. Also, any offline calibration of a gain compensation can beeliminated, which saves time/resource cost in camera production.

For further explanation, FIG. 5 sets forth a flow chart illustrating anexample method of providing enhanced phase detection in an imagecapturing device having phase detection autofocus devices in accordancewith some implementations of the present disclosure. The example methodof FIG. 5 includes extracting 510 pixel data from image data. The imagedata is captured from an image capturing device having a PDAF sensor.For example, an image can be captured from the image capturing device110 of FIG. 1 using the components of the camera module 111 such as thelens 123 and the PDAF sensor 125. The PDAF sensor can be used, inconjunction with the lens 123, to focus on an object of interest. ThePDAF sensor 125, equipped with phase detection (PD) capabilities, canseparate left and right light rays though the lens 123 in order tocapture or ‘sense’ left and right images. The PDAF sensor 125 extractsthe PDAF pixel data from the image data. In an additional aspect, thePDAF sensor 125, in association with the processor 102 (which can be animage signal processor) and the APD 116, extracts the PDAF pixel datafrom image data.

The example method of FIG. 5 also includes extracting 520 one or morefeatures from the extracted pixel data. In the method of FIG. 5 ,extracting 420 features also includes removing 521 irrelevant pixeldata. Removal of irrelevant pixel data is described below in greaterdetail. The PDAF sensor 125 can carry out the extraction process. Also,in some operations, the PDAF sensor 125, in association with theprocessor 102 (which can be an image signal processor) and the APD 116,can carry out the extraction of the features from the pixel data.

In one aspect, the operations for extracting the features from the pixeldata can include, for example, performing an edge detection operation.Such edge detection can be performed on the left pixel data and theright pixel data acquired form the image capturing device 110 byidentifying points at which the brightness of the left pixel data andthe right pixel data changes or discontinues. The edge detection detectsedge points on the left pixel data and the right pixel data. Morespecifically, the edge detection operation identifies regions of greaterintensity variations are located such as, for example, the edge pointsof the left pixel data and the right pixel data. These points where thebrightness changes can be organized into a set of line segments that canbe referred to as “edges.” The information on a detected edge of theleft pixel data and the right pixel data is applied to a specific targetobject of the left pixel data and the right pixel data. In order toacquire the edge (e.g., edge point) data, a colormap can be used. Thecolor map can be sorted in increasing order according to one of avariety of criteria such as, for example, hue variation, saturationvariation, and intensity variation. Thus, the features extracted fromthe pixel data can be performed by edge detection.

The example method of FIG. 5 also includes determining 530 a phasedifference between the one or more features of the pixel data extractedfrom the image data. In one aspect, the PDAF sensor 125 compares theextracted features of the pixel data, which was extracted from the imagedata. The difference in extracted feature image positions is used todetermine a shift or phase difference to enable autofocus of the imagecapturing device 110 to determine the magnitude and direction of themovement of lens 123 for bringing a target image into focus. Thus, theextracted features (forming a pair of extracted features) from the leftpixel data such as, for example, the left feature image 340 can becompared to the extracted features from and the right feature image 342to determine the phase difference information or “true phasedifference.” In another aspect, the pair of extracted features (e.g.,left feature image 340 and right feature image 342) can be referred toas phase detection extracted feature phase detection pixels.

As mentioned above, extracting features from pixel data includes removal521 of irrelevant pixel data. For further explanation, therefore, FIG. 6sets forth a flow chart illustrating an example implementation ofremoving the irrelevant pixel data in accordance with someimplementations of the present disclosure. In the method of FIG. 6 ,removing 521 irrelevant pixel data includes filtering 610 the pixel dataextracted from the image data to remove the irrelevant pixel data. Inone aspect, the filter 124, in association with processor 102, filtersthe phase detection pixels extracted from the image data to remove theirrelevant pixel data. The filter 124 performs the filtering using a1-dimension (1D) high/band-pass filter to remove the irrelevant pixels.In another aspect, the filtering can occur using other types of filters.The irrelevant pixels can be noise or weak pixels for low-contrast orlow-textured images or small, irrelevant objects. Also, the irrelevantpixels can be pixels with imperfect compensation, flat areas, orhorizontal lines and can be removed to eliminate any bias matchingresult. In another aspect, the irrelevant pixels are those pixels thatare not extracted identified using an edge detection operation. In oneaspect, upon the phase detection pixels being extracted from the imagedata, the filter 124, for example, filters the irrelevant data. Inanother aspect, the filter 124, for example, filters the irrelevant datain conjunction with extracting the one or more features from the phasedetection pixels.

For further explanation, FIG. 7 sets forth a flow chart illustrating anexample implementation of extracting pixel data from image data inaccordance with some implementations of the present disclosure. In themethod of FIG. 7 , extracting 510 the pixel data also includesextracting 710 left pixel data and right pixel data from the image data.For example, the image capturing device 110 extracts left PD pixels 320and right PD pixels 324 from the image data 310 (e.g., a full rawimage). In one aspect, the PDAF sensor 125 extracts left pixel data andright pixel data from a raw image where the raw image is an image filecontaining minimally processed data from the image capturing devices.That is, the PDAF sensor can separate left and right light rays thoughthe lens 123 of the image capturing device 110 to sense left and rightimages.

Also in the method of FIG. 7 , extracting 520 features from the pixeldata includes extracting 720 a feature from the left pixel data and afeature from the right pixel data prior to determining the phasedifference. Such extraction of a feature is described above with respectto various edge techniques, but is applied here separately to the leftand right pixel data.

The example method of FIG. 7 also includes determining 730 the phasedifference between a feature extracted from left pixel data and afeature extracted from right pixel data in an image feature domain. Inone aspect, the PDAF sensor 125, in association with the filter 124,determines a true or enhanced phase difference between the left extractfeature image and a right extracted feature image without the irrelevantpixel data. This true or enhanced phase difference between the leftextract feature image and a right extracted feature image can be used todetermine a shift of a lens 123 for autofocus. By determining the truephase difference, which is determining the phase difference without theresults being compromised by irrelevant pixel data, a left/right gaincompensation is eliminated. The PDAF sensor 125 can, for example,determine a more accurate direction and more precise amount of positionmovement of the lens 123. In another aspect, the PDAF sensor 125 can,for example, more accurately determine the phase shifts between the leftextract feature image and a right extracted feature image to determinespatial characterization or conversion values to convert the phaseshifts to lens offsets.

Once the true phase difference is determined, a matching operation canbe carried out. To that end, FIG. 8 sets forth a flow chart illustratingan example implementation of matching features of the pixel data inaccordance with some implementations of the present disclosure. Theexample method of FIG. 8 includes matching 810 the one or more featuresof the pixel data extracted based on the phase difference. Such matching810 can be carried out by performing 820 left-right matching between afeature extracted from left pixel data and a feature extracted fromright pixel data in the image feature domain. In one aspect, the PDAFsensor 125, in association with the filter 124, matches the leftextracted feature image and a right extracted feature image without theirrelevant pixel data (as the irrelevant data was removed 521). That is,a left/right matching operation can be performed in the image featuredomain to determine a more accurate phase difference (a true phasedifference) following the pre-processing operation of featureextraction, which also occurs in the image feature domain. In this way,by using the pre-processing operation in an image feature domain, anyneed for per-pixel gain compensation and offline calibration of gaincompensation is eliminated.

In view of the foregoing, readers of skill in the art will appreciatethat implementations in accordance with the present disclosure offer anumber of advantages. Implementations provide applications or operationsof an image capturing devices to execute a pre-processing operation inan image feature domain to eliminate the need for per-pixel gaincompensation and offline calibration of gain compensation. In this way,the user experience is improved.

Implementations allow memory-local computing to be used efficiently foratomic operations, which can improve performance for a range ofimportant workloads (e.g., graph analytics, sparse matrix algebra,machine learning, etc.). Such applications can take advantage of cachelocality when available, and dynamically identify coalescingopportunities to enable more efficient multi-module memory-localprocessing operations.

Implementations can be a system, an apparatus, a method, and/or logiccircuitry. Computer readable program instructions in the presentdisclosure can be assembler instructions, instruction-set-architecture(ISA) instructions, machine instructions, machine dependentinstructions, microcode, firmware instructions, state-setting data, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. In some implementations, electronic circuitryincluding, for example, programmable logic circuitry, field-programmablegate arrays (FPGA), or programmable logic arrays (PLA) can execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and logic circuitry according to some implementations of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bylogic circuitry.

The logic circuitry can be implemented in a processor, otherprogrammable data processing apparatus, or other device to cause aseries of operational steps to be performed on the processor, otherprogrammable apparatus or other device to produce a computer implementedprocess, such that the instructions which execute on the computer, otherprogrammable apparatus, or other device implement the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and logic circuitry according to variousimplementations of the present disclosure. In this regard, each block inthe flowchart or block diagrams can represent a module, segment, orportion of instructions, which includes one or more executableinstructions for implementing the specified logical function(s). In somealternative implementations, the functions noted in the block can occurout of the order noted in the figures. For example, two blocks shown insuccession can, in fact, be executed substantially concurrently, or theblocks can sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustrations, and combinations ofblocks in the block diagrams and/or flowchart illustrations, can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts or carry out combinations of special purposehardware and computer instructions.

While the present disclosure has been particularly shown and describedwith reference to implementations thereof, it will be understood thatvarious changes in form and details can be made therein withoutdeparting from the spirit and scope of the following claims. Therefore,the implementations described herein should be considered in adescriptive sense only and not for purposes of limitation. The presentdisclosure is defined not by the detailed description but by theappended claims, and all differences within the scope will be construedas being included in the present disclosure.

1. A method of providing enhanced phase detection, the methodcomprising: extracting pixel data from image data, the image datacaptured from an image capturing device having a phase detectionautofocus (PDAF) sensor; extracting one or more features from the pixeldata, including removing irrelevant pixel data, wherein removingirrelevant pixel data further comprises filtering the pixel dataextracted from the image data using one or more of a high-pass filter ora band-pass filter; and determining a phase difference between the oneor more features.
 2. The method of claim 1, wherein the high-pass filteror the band-pass filter comprises a 1-dimensional filter.
 3. The methodof claim 1, wherein extracting the pixel data from the image datafurther comprises extracting left pixel data and right pixel data fromthe image data.
 4. The method of claim 3, wherein extracting the one ormore features from the pixel data further comprises extracting a featurefrom the left pixel data and a feature from the right pixel data priorto determining the phase difference.
 5. The method of claim 1, whereindetermining the phase difference further comprises determining the phasedifference between a feature extracted from left pixel data and afeature extracted from right pixel data in an image feature domain. 6.The method of claim 1, further comprising matching the one or morefeatures of the pixel data based on the phase difference.
 7. The methodof claim 1, further comprising performing left-right matching between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain.
 8. An apparatus forproviding enhanced phase detection, the apparatus comprising a computerprocessor, a computer memory operatively coupled to the computerprocessor, the computer memory having disposed therein computer programinstructions that, when executed by the computer processor, cause theapparatus to carry out: extracting pixel data from image data, the imagedata captured from an image capturing device having a phase detectionautofocus (PDAF) sensor; extracting one or more features from the pixeldata, including removing irrelevant pixel data, wherein removingirrelevant pixel data further comprises filtering the pixel dataextracted from the image data using one or more of a high-pass filter ora band-pass filter; and determining a phase difference between the oneor more features.
 9. The apparatus of claim 8, wherein the high-passfilter or the band-pass filter comprises a 1-dimensional filter.
 10. Theapparatus of claim 8, wherein extracting the pixel data from the imagedata further comprises extracting left pixel data and right pixel datafrom the image data.
 11. The apparatus of claim 10, wherein extractingthe one or more features from the pixel data further comprisesextracting a feature from the left pixel data and a feature from theright pixel data prior to determining the phase difference.
 12. Theapparatus of claim 8, wherein determining the phase difference furthercomprises determining the phase difference between a feature extractedfrom left pixel data and a feature extracted from right pixel data in animage feature domain.
 13. The apparatus of claim 8, further comprisingcomputer program instructions that, when executed, cause the apparatusto carry out matching the one or more features of the pixel data basedon the phase difference.
 14. The apparatus of claim 8, furthercomprising computer program instructions that, when executed, cause theapparatus to carry out performing left-right matching between a featureextracted from left pixel data and a feature extracted from right pixeldata in an image feature domain.
 15. A non-transitory computer readablemedium for providing enhanced phase detection, the non-transitorycomputer readable medium comprising computer program instructions that,when executed, cause a computer to carry out: extracting pixel data fromimage data, the image data captured from an image capturing devicehaving a phase detection autofocus (PDAF) sensor; extracting one or morefeatures from the pixel data, including removing irrelevant pixel data,wherein removing irrelevant pixel data further comprises filtering thepixel data extracted from the image data using one or more of ahigh-pass filter or a band-pass filter; and determining a phasedifference between the one or more features.
 16. The non-transitorycomputer readable medium of claim 15, wherein the high-pass filter orthe band-pass filter comprises a 1-dimensional filter.
 17. Thenon-transitory computer readable medium of claim 15, wherein extractingthe pixel data from the image data also includes: extracting left pixeldata and right pixel data from the image data; and extracting a featurefrom the left pixel data and a feature from the right pixel data priorto determining the phase difference.
 18. The non-transitory computerreadable medium of claim 15, wherein determining the phase differencefurther comprises determining the phase difference between a featureextracted from left pixel data and a feature extracted from right pixeldata in an image feature domain.
 19. The non-transitory computerreadable medium of claim 15, further comprising computer programinstructions that, when executed, cause the computer to carry outmatching the one or more features of the pixel data based on the phasedifference.
 20. The non-transitory computer readable medium of claim 15,further comprising computer program instructions that, when executed,cause the computer to carry out performing left-right matching between afeature extracted from left pixel data and a feature extracted fromright pixel data in an image feature domain.